Variational Latent-State GPT for Semi-Supervised Task-Oriented Dialog Systems

Recently, two approaches, fine-tuning large pre-trained language models and variational training, have attracted significant interests, separately, for semi-supervised end-to-end task-oriented dialog (TOD) systems. In this paper, we propose Variational Latent-State GPT model (VLS-GPT), which is the first to combine strengths of approaches. Among many options models, generative inference learning TOD system, both as auto-regressive based on GPT-2, can be further trained over a mix labeled unlabeled data in manner. training VLS-GPT statistically computationally more challenging than previous works sequential latent variable use turn-level first-order Markovian. The non-Markovian due Transformer architecture. work, establish Recursive Monte Carlo Approximation (RMCA) objective with prove its unbiasedness. Further, develop computational strategy sampling-then-forward-computation realize RMCA, successfully overcomes memory explosion issue using speeds up training. Semi-supervised experiments are conducted benchmark multi-domain datasets different languages - MultiWOZ2.1 CrossWOZ. shown significantly outperform supervised-only self-training baselines.